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Assessing the Impact of the National Science Foundation’s Urban Systemic 
Initiative: The Impact of Policy on Student Achievement and Closing the Gap in Four 

USI Sites 

Abstract 

In this paper we describe the data sources and methods used in conducting our 
three-year evaluation of NSF’s Urban Systemic Initiative in four cities, provide a 
discussion of two approaches to analyzing our results; structural equation modeling 
(SEM) and a complementary analysis of the standardized means of mathematics 
achievement across years, and finally, examine the results of these analyses as they 
address the question of what factors or indicators increase student achievement and close 
the achievement gap most effectively. Results of SEM and Path analysis support the 
addition of a seventh driver, school culture to the NSF sis driver model. Analysis of 
mathematics achievement of the schools in our study indicates the achievement gap has 
been reduced. In addition, our analyses using comparison non-USI sites show that 
compared to relatively affluent, predominantly White schools, USI schools have made 
great progress in closing the gap. 
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The NSF Six-Driver Model of Urban Systemic Reform: The Impact of Policy on Student 
Achievement and Closing the Gap in Four USl Sites 

Introduction 

Since the early 1980’s urban schools have been the target of reform, for the most part 
piecemeal, aimed at one or two dimensions of the schooling process. NSF’s approach, in 
contrast to many plans preceding it, emphasizes the simultaneous application of a number of 
policy levers (called drivers). Additional guiding assumptions argue that all children can achieve 
to high academic standards and that professional development of teachers in the context of 
school-based learning communities is central to the reforms. To improve educational 
opportunities for urban students and to strengthen teaching and learning in urban schools, the 
strategy in question must be designed to (1) promote the academic achievement of all students 
and to close the achievement gap, (2) engage teachers and school principals in forging a 
community of learners, and (3) involve parents and stakeholder groups, or it will not be 
sustainable. 

In this paper we will describe data sources and methods used in conducting our three- year 
evaluation of NSF’s Urban Systemic Initiative in four cities, provide a discussion of two 
approaches to analyzing our results: structural equation modeling (SEM) and a complementary 
analysis of the reform group levels’ standardized means across years, and finally, examine the 
results of these analyses as they address the question of what factors or indicators increase 
student achievement and close the achievement gap most effectively. 

Related literature 

Urban systems enroll more than a third of all students attending public school in the 
United States (www.nsf gov). Despite past reform efforts involving urban schools, the gap in . 
academic achievement persists between urban students and their counterparts in suburban and 
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rural school systems, especially in science and mathematics. The National Science Foundation’s 
Urban Systemic Initiatives (USI) program was designed to focus on the largest cities with 
greatest numbers of students living in poverty. Beginning in 1993, twenty-one urban sites were 
granted funding from NSF in four yearly cohorts through 1996, each site receiving a five year 15 
million dollar award. The ethnic and racial diversity apparent across the range of USI urban sites 
reflects NSF’s desire to reach out to the most underserved students by providing catalytic 
funding to their school districts to promote increased achievement for all students in 
mathematics, science, and technology. 

The National Science Foundation (NSF) has been critical in providing both the support 
and the conceptual rubric for institutionalizing systemic reforms. According to the NSF, 
Systemic reform occurs when all essential features of schools and school systems 
are engaged and operating in concert; when policy is aligned with a clear set of 
goals and standards; when forthcoming improvements and innovations become 
intrinsic parts of the ongoing educational system for all children; and when the 
changes become part of the school system’s operating budget (NSF, 2000). 

Initially, NSF supported 25 states undertaking reform through its Statewide Systemic Initiatives 
program (SSI). The goal of the SSI is to assist the states in developing the capacity to move from 
independently devised science and mathematics educational reform measures to state-developed 
efforts. Coordinating such improvements as teacher preparation, the development of standards- 
driven instructional materials, and the assessment of student performance cover the range and 
scope of the SSI to date (Westat* McKenzie Consortium, 1998, October, p. 6). Although large 
numbers of teachers received curricular and instructional materials, and schools of education 




5 



Closing the Gap, 3 



undertook some degree of curriculum change in teacher preparation programs, these programs 
failed to reach those urban schoolchildren in the most difficult circumstances. 

NSF established the Urban Systemic Initiatives (USI) program in 1993. Funding under 
the USI program was made available to urban school districts in the U.S. with the highest rates 
of poverty among their school-aged children according to the 1990 census. Of 28 eligible school 
districts, 22 successfully applied for and received a total of 15 million dollars each over a four- 
year period to carry out systemic reforms in math and science (Westat*Mckenzie, 1998). 
NSFfunding was considered a medium for developing, expanding, or sustaining reform through 
partnerships with businesses, educational institutions, and community organizations. The 
University of South Florida’s evaluation project targeted four of the cities receiving funding 
under the USI: Miami, Memphis, El Paso and Chicago. 

School Culture, a Seventh Driver? 

School culture can be viewed as the mediating set of factors that influence the creation of 
social ties and relationships, and may be the critical element enhancing or curtailing effective 
teaching and successful student outcomes. It is increasingly seen as a critical element in 
supporting or defeating school reform (Fullan, 2001). Additionally, there is evidence to support 
the notion that students learn better when principals, teachers and others develop collaborative 
relationships within a professional learning community (Newman & Wehlage, 1995; Louis, 
Marks, & Kruse, 1996; Stein, 1998). According to Supovitz and Turner (2000), teachers who felt 
supported by their principal reported significantly greater use of reform approaches than did 
teachers who did not feel encouraged by their school leader. 
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The Ciirrent Crisis: Closing the Persistent Achievement Gap 

While some researchers and reformers have successfully carried out enduring reform at 
the school level, the challenge still remains to create sustainable systemic reform across all 
school districts, regions and states. The achievement gap persists between the genders, as well as 
among ethnic group members and socioeconomic levels. Corresponding gaps are also evident in 
financial inequities in school funding; schools that cannot keep up with high technology in either 
equipment or teaching; tracking and course offerings based on more and less challenging 
academic content; and the challenges of high stakes testing. 

So, what precisely is the crisis? It is a combination of all these things and their effects on 
the most important piece of the puzzle: student outcomes. The United States requires a literate, 
technologically savvy, mathematically proficient population capable of high-level problem 
solving. Students in turn require an environment in which teachers can teach the materials 
effectively and challenge students to think creatively while instilling appropriate knowledge and 
skills as a foundation for future learning. 

The first issue to acknowledge is the persistent educational achievement gap between 
minority and non- minority students. In addition, the gap between poor and affluent students and 
the gender gap also persist when we consider certain achievement markers. Although these gaps 
are well documented in the United States, no clear conclusion has been reached to explain tteir 
persistence, while many “measures” showing disparities in cognitive abilities between groups 
have been discredited as valid indicators of achievement differences. 

NSF-sponsored USI reform efforts attempt to reduce achievement gaps and address 
factors contributing to eliminating disparities in achievement. First, low performing schools 
often suffer from a lack of coherence among activities deemed as priorities by school district or 
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schooHevel administrators. Bryk and his colleagues (1998) in Chicago refer to these as 
“Christmas tree schools.” The lack of coherence includes fragmentation of the curriculum, 
fragmentation or lack of coordination in organizing the school day, poorly related or 
incompatible instructional strategies, inconsistent behavioral expectations, and the lack of a 
shared purpose and shared values. 

Second, many schools face a poorly organized or non-existent program to support the 
acquisition of knowledge and skills needed to engage in effective reform at the school level. In 
addition, teachers and principals lack the time and resources to mobilize the information, skills 
and knowledge to undertake radical transformations of teaching, learning, assessment, and 
school organization. Third, there are major disincentives for teachers to elect to work in schools 
with histories of failure much less to stay with schools undergoing major shifts in practice. 
Developing pedagogical skills, the capacity to undertake complex organizational transformation, 
and the willingness to stay with a career with minimal financial rewards for the work required 
seems a most difficult challenge for even the most dedicated teachers and school staff 

Measures such as standardized testing, increased school accountability, community 
involvement, and resource allocation must be structured and evaluated in ways that relate them 
to achievement differentials. An increasingly important factor is the use of technology as an 
important part of school and work environments. When business leaders worry about applicants 
meeting their minimum literacy requirements and having higher level thinking skills, the 
question of educational efficacy come to the forefront (Mizell, 1992, p. 50). The commitment of 
educational leaders to increase the achievement levels of all students should be a real part of 
reform efforts to make certain all students are ready and able to either continue into higher 
education or be successful in a rapidly changing, technically demanding job market. Systemic 
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changes in both structure and practice are important to this goal and to the reduction of 
achievement differentials through increased alignment of teaching philosophy and policies with 
classroom practices. 

Methods and Results 

Description of SEM and the Indicators 

Our complex array of studies and research questions allowed us to develop multiple 
indicators for each of the NSF drivers, to do so over a three-year period, and ultimately to 
execute a comprehensive analysis of multiple salient features fostering or inhibiting student 
attainment. In turn, these indicators informed our conceptualization of the NSF driver model 
including a hypothesized seventh driver. Indicators included both organizational and individual 
level factors such as student achievement and student engagement (Drivers 5 and 6); teachers’ 
reports of their professional development experiences, use of technology in the classroom, 
involvement in decision-making processes, etc (Drivers 1 and 3); school district assessment 
practices and school level support structures (Driver 2); teachers’ classroom practices (Driver 1); 
the nature of community-school partnerships and other arrangements with business and industry 
(Driver 4); school climate and school leadership (Driver 3 and the hypothesized Driver 7). A 
variety of statistical analyses were conducted using SAS, Version 8.2 (SAS Institute, 2001). 
Descriptive statistics, correlations, internal consistency, reliability, and path analysis were 
calculated. For the analyses reported here related to our Structural Equation Models and Path 
Analysis techniques, we computed student math achievement gains over the period of the reform 
(1995-1999). The gap was conceptualized in terms of both gender and race/ethnicity (Hispanic, 
Black, and White). Analyses reported here are for students, teachers, administrators, and 
documents in Miami-Dade only. 
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Analyses Using SEM and Path Analysis 

A series of principal component analyses were conducted to determine the structure of 
our instrumentation and to further reduce our set of variables to a conceptually meaningful yet 
smaller subset. For example, the Study of the Enacted Curriculum: Survey of Classroom 
Practices in Mathematics and Science (SEC-T) (Blank, 2001) is a 155-item teacher survey with a 
companion measure for students (SEC-S) with 53 overlapping items. Our principal component 
analyses yielded a smaller number of factors, five altogether. Each instrument was subjected to a 
principal components factor analysis and items with loadings greater than 0.40 were averaged to 
create factor scores. A comprehensive list of components, the underlying factors, and the original 
items and estimates of factor reliability are presented in Table 1. 

Principal Components Analysis . Components scores were generated using the factor 
score estimates from our analyses matched with the associated NSF driver. For example. Driver I 
in the NSF model includes standards-based instruction. As mentioned previously, we identified 
five salient factors from the Study of the Enacted Curriculum: Survey of Classroom Practices 
and scored teachers by averaging their ratings on the higher loading items. We used teacher 
classroom item- means to generate factor scores on the student version of the SEC. Altogether 
our data included student and teacher reports of classroom instructional activities, in addition to 
rubric scores from classroom observations, principal interviews, and school improvement plans. 
Finally, we computed factor scores that represented traditional teacher beliefs, standards-based 
teacher beliefs, aspects of school culture and student achievement. There were 52 identified 
factor scores, and each driver was subjected to a principal components analysis using the 
associated factor scores. We used these component scores to generate the correlation matrix 
used in the path analysis discussed below. 
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Path Analytic Approach . Path analysis expresses regression equations in the form of 
causal diagrams to portray complex relationships among independent and mediating variables 
(e.g., stakeholders, resources, and polices) that in turn explain dependent variables (e.g., 
mathematics achievement and standards-based instruction) expected to impact the achievement 
gap in the context of systemic reform (Pedhazur, 1982). Path analysis allows us to estimate the 
indirect effects of policies on achievement as well as the direct effects of instruction on 
achievement. As discussed previously, our study involved a variety of measures yielding 17 
components representing broad categories of variables of interest. In the path analysis, SAS 
PROC CALIS was used to determine the contribution of the exogenous variables (e.g., policy, 
resource, and stakeholder drivers) on the endogenous variables (e.g., standards-based instruction 
and achievement outcome drivers). This analysis also determined the relative contribution of the 
mediating standards-based instructional variables on student achievement. All paths in Figure 1 
represent direct effects or the beta coefficients from multiple regression analysis. These path 
coefficients indicate the relative contribution of each driver. In this section we discuss the 
relationships among the component scores first descriptively and then followed by the results of 
our path analysis. 

Correlations among standardized component scores (as reported in Table 2) suggest two 
areas for discussion. First, the component scores represent independent measures of a conceptual 
group or driver. Standards-based instruction has five components with modest inter-correlations. 
In this group the strongest relationship is an inverse relationship between performance projects 
and subject-centered class work (i=-.24). Other relationships are slightly positive or slightly 
negative and are in the 0.1 range in magnitude. Two policy-related components (instructional 
influences and teacher accountability) have a slight inverse relationship suggesting that the two 
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policy components are independent of each other (f=-. 06). Relationships are modest yet positive 
among the unified resources grouping. Equity and professional development coursework were 
correlated (r=.24) and coursework and instruction were similarly correlated (r=.21). Strongest 
correlations are demonstrated for achievement gap related components (.48), while the 
remaining correlations were between -0.02 and 0.17. Correlations greater in magnitude than 
0.35 are significant at the .05 level. 

Finally the correlations among component scores in different conceptual groups were 
somewhat mixed. Technology projects conceptual group was positively correlated with each of 
the unified resources’ indicators including equity (.41), professional development coursework 
(.51), and professional development related to standards-based instruction (.34). There were 
strong correlations between each of the achievement gap indicators and the community 
stakeholder indicator. The component score related to Hispanic, White, and female students was 
correlated .77 with the community- stakeholder indicator while the male and Black indicator was 
correlated with the community- stakeholder indicator almost as strongly (.68). 

The components analysis led to two models for understanding changes in student 
achievement as delineated in Tables 3a and 3b. In addition. Figure 1 presents a path model of 
student achievement gains discussed following our discussion of the six driver model. Table 3b, 
also discussed below, presents information on all seven drivers included in our research and 
shows effects on the achievement gap for male and students ale students. Each model was re- 
estimated after paths with coefficients smaller than .05 were removed. 

Gains in Student Achievement: NSF Six Driver Model. The NSF six driver model (as 
shown in Table 3a) explained the association among aspects of standards based instruction 
(classroom instruction, technology projects, performance projects, student projects, and class 
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work), policy alignment (student, parent and principal views on professional development), and 
resources related to professional development. It explained 16.6 percent of the variability in the 
mathematics achievement component. This could be attributed to three of the component groups 
- performance projects, technology projects, and standards-based practice observed in sample 
classrooms. A one standard unit change in performance projects results in a .23 standard unit 
change in student achievement gains over the period of the reform. A one-unit change in 
technology projects results in a -. 1 1 standard unit change in achievement suggesting that students 
of teachers who use technology in their classrooms had smaller achievement gains than those 
whose students were required to demonstrate their knowledge. Finally, a one unit change in 
observations of standards-based practice component results in a .10 standard unit change in 
student achievement. The process drivers had small indirect effects on student mathematics 
achievement gains when direct effects and total effects are examined. A one standard unit 
change in the process component grouping results in a fractional standard unit change in the 
outcome component group, student achievement gains. Direct effects were found for the policy 
component related to principals’ view of the impact of professional development (.22), equity (- 
.25), time in professional development (.11) and professional development involving standards 
based activities (.15). The total effect of instructional influences on achievement was positive 
(.10) while the total effect of stakeholders/community on achievement was negative (-.09). 

We also modeled through path analysis an achievement model in which the components’ 
content \\as largely comprised of gains for Hispanic, White and female students (See Figure 1). 
This model, which explained 72.8 percent of the variability in student achievement, may be 
contrasted with a model whose components were related to score gains for Black and male 
students (r^=.58). This suggests different explanations for reducing differences in student 
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groups. For this second model using success in reduction of the achievement gap as the outcome 
measure we report the following direct effects. The unit change in the technology projects 
component group results in a .33 standard unit change in the achievement gap. A unit change in 
performance projects leads to a .20 standard unit change in the achievement gap while a standard 
unit change in student projects results in a .10 standard unit change in the achievement gap. 
Finally a standard unit change in classroom observed standards based practice results in a .06 
standard unit change in the achievement gap. 

7 Driver Student Achievement Model. A comparable model, hypothesizing a seventh 
driver, school culture, as shown in Table 3b explained 19.9 percent of the variance associated 
with gains in student achievement. In a revision of the original NSF model, this additional 
process was thought to affect instruction and thus student achievement. Four instructional 
components of the model had path coefficients larger than .05, technology projects (-.13), 
performance projects (.28), student projects (.09), and classroom observations of standards 
based- instruction (.06). These direct effects suggest that a one standard unit change in the use of 
performance projects in the classroom is associated with a .28 standard unit change in 
achievement gains. The other instructional components (subject centered representations) had 
smaller and negative coefficients suggesting that these elements had small, but less undesirable 
effects on student achievement. The school culture components present the following changes in 
achievement in interpreting their path coefficients. A standard unit change in teachers’ opinions 
(e.g., a culture of sharing, beliefs about standards) results in a .2 1 standard unit change in student 
achievement gains. Finally, a unit standard change in the remaining school culture variable 
summarizing vision, teamwork, facilitative leadership and learning community results, in a .08 
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unit change in the achievement gap. In sum, the addition of this driver adds to the explanatory 
power of the NSF reform model. 

Analysis using Comparison Non-USI sites 

We also opted to tackle the ‘‘gap issue” in another way. In addition to the multiple 
indicators for each school in our four participating sites used in the prior analyses, we were also 
fortunate to obtain access to an NCES public use data set with achievement and related data for 
virtually all schools in the U.S. Because USI schools draw their enrollments from predominately 
low- income neighborhoods with large numbers of ethnic minority residents, we selected 
primarily majority non-USI schools from the geographical areas closest to our sites. We defined 
primarily majority schools as having a student population at least 55% White. Further, we 
retained only those non-USI schools for which data were available the four years of interest. For 
Chicago, we selected 37 schools from the suburban cities surrounding Chicago, including 
Northbrook, Oak Park, and Oak Lawn. These inner-ring suburban sites are relatively affluent 
compared to central city Chicago. For El Paso, we were unable to locate enough majority 
schools in the immediate surrounding area. Texas is divided into 20 education regions, and El 
Paso is located in region 19, the western- most region of the state. We located 12 schools in 
region 15, which is near the center of Texas and includes the city of San Angelo, and we located 
17 schools in region 20, which is southeast of region 19 and contains the city of San Ajitonio. In 
the case of Memphis, we used a proxy variable in our decision rule due to missing information 
on racial makeup of schools in our NCES database. We used 23 schools with 55% or less on 
free or reduced lunch as our cut point, and selected these schools from the Shelby County School 
District. While Memphis is located in Shelby county, the Memphis City School District is a 
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separate entity from the Shelby County School District. For Miami, we located 93 schools in 
Pinellas County, which is in the central western portion of the State. 

As mentioned, at issue here is the analysis of the achievement gap between primarily 
minority and primarily majority schools. Specifically, we measured math achievement at the 
school level, and, in order to facilitate comparison across the districts, we standardized math 
achievement within district (i.e., we converted the scores to within-district Z scores). 
Comparability of the resulting scores rests on the assumption that although different instmments 
were used, each instrument is an adequate measure of the same constmct (math achievement). 

In making our comparisons, we first combined across school level (i.e., elementary, 
middle, and high school) and completed a sort of “omnibus” comparison between our USI and 
norhUSI Schools. This analysis is depicted in Figure 2 with 40 studied USI schools and 182 
norhUSI schools. Note that the dotted line represents non-studied USI schools. Because USI 
grants were made to participating districts, one could consider all schools in our four districts as 
USI schools. Thus, the dotted line represents all schools in each district less our studied USI 
schools. In Figure 2 we see that our studied USI schools are over one-third a standard deviation 
below our non-USI Schools at the baseline. However, over the four years, the USI schools close 
that gap to just over one tenth of a standard deviation. Our repeated measures analysis of 
variance yielded no main effect of time, and a borderline main effect for group (i.e., studied USI 
vs. norhUSI) F=3.56 (1,220), p. = .06. Time by group interaction was also a borderline effect, F 
= 2.15 (3,660), p = .10. Further analyses revealed that differences between groups in 1996 and 
1997 were significant (p < .05), while the differences between the groups in 1998 and 1999 were 
not, indicating movement toward closing the gap. It is also reasonable to assume that there 
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would be differences in the gap and changes across time by school level. Let us now turn our 
attention to analyses at the different school levels. 

Figure 3 shows comparisons between the studied USI (N=17) and non-USI (N=106) 
schools at the elementary level. We see a very similar pattern to Figure 2. In 1996 there is a gap 
of approximately one half standard deviation, which reduces each year until 1999 where it is 
almost zero. We also ran a repeated measures analysis of variance here. We found a main 
effect for group, F = 4.03 (1,122), p = .05, but no main effect for time. Our interaction between 
time and group was borderline significant, F = 2.5 1 (3, 366), p = .07, with further analyses 
revealing that the difference between the groups was significant for all years (p < .05) but 1999, 
signahng the closing of the gap. 

In Figure 4 we see comparisons between studied USI (N=15) and non-USI (N=35) 
schools at the middle school level. While the gap does close somewhat, it appears that most of 
this closure is attributable to falling achievement levels recorded for non-USI schools. 
Additionally, the studied USI schools trend upward from 1996 to 1997, but then fall, never 
regaining their peak. A repeated measures analysis of variance revealed no significant effects, 
which may in part result from our low power and unbalanced sample sizes. 

Figure 5 reveals yet a different pattern. Of interest here is the fact that, studied USI high 
schools begin and end above the mean (i.e., above zero). This set of results is similar to Figure 
2, but with studied USI (N=8) and non-USI (N=41) schools gaining two-tenths of a standard 
deviation. And, again, the story is positive. The gap in 1996 is approximately two-tenths of a 
standard deviation and ends in 1999 at just over one-tenth of a standard deviation. A repeated 
measures analysis of variance revealed no significant effects, which, again may be due to our 
low power and our rather unbalanced sample sizes. Our analyses here portray the success of the 




17 



Closing the Gap, 15 



Urban Systemic Initiative in closing the gap. Not all findings were statistically significant, but, 
practically, they are very positive, representing a step in the right direction. 

Discussion 

The current thrust for systemic reform addresses the assumption that schools have not 
provided students, especially those students attending the least academically successful schools, 
with knowledge necessary to be successful in society - the outcome stressed by Newmann and 
Wehlage (1995). Students were not sufficiently challenged by the instruction they received with 
the result that many were ill- prepared to attend college upon graduation, enter technologically 
complex careers, or engage in challenging intellectual work. By setting more rigorous standards 
for students, the general level of student achievement would rise, better preparing students for 
post-secondary educational opportunities and employment (Roeber, 1999). Our overall analysis 
of mathematics achievement of the schools in our study indicates the achievement gap has been 
reduced. In addition, our analyses using comparison non-USI sites show that compared to 
relatively affluent, predominantly White schools, USI schools have made great progress in 
closing the gap. These analyses also suggest that it is difficult to sustain an upward trend in 
achievement at the middle school level. 

Systemic reform differs from past efforts in emphasizing rigorous academic coursework 
for^ students. In addition to improving the overall quality of education for all children, as 
underscored in the effective schools approach, educational equity is at the core (Kahle, 1998; 
Smith & O’Day, 1991; O’Day & Smith, 1993). As a result, achievement gaps are expected to 
decrease (Williams, 1996). Systemic reform recognizes that attempts to change one aspect of the 
system will require chants in other aspects at all levels of the system. However, the most 
important change must occur at the level of the school classroom, buttressed by policies at the 
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school district and state levels, especially policies focused on curriculum, instruction, and 
assessment. Our results add texture and complexity to our knowledge of classroom reforms, 
showing, for example, that factors such as technology in the classroom have a strong impact on 
achievement for many Hispanic and White students and may not leverage similar gains for 
Blacks and males. We also learned from these analyses that students whose teachers using 
performance projects, technology projects, and standards-based practice experience gains in 
math achievement. In addition, community stakeholders’ involvement in schools positively 
affect student achievement. An obvious policy lesson here is that poorly performing schools 
must gamer support from various constituencies to improve student outcomes. Our models 
indicate that another important aspect of improved student outcomes is the school culture or 
learning community environment. In schools where teachers view themselves as learners and 
believe that their students can achieve, improved student outcomes are likely to result. 
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Table 1. Results of principal component analysis for Miami sample 



Driver Driver Name Factor Name 

Driver 1 Standards- Based Instruction 

Driver 1 . 1 Classroom Observation of SB Instruction 

Modeling 
Connections and 
Reasoning 
a=.94* 

Connections 

a=.86 

Student Centered 
a=.58 

Problem Solving 
a=90 

[Homework] 

Driver 1.2 Technology Projects 

Use Calculators 
a=.86 

Doing SB 
Activities 
a=70 
Worksheet 

[Hands-on]* 

Driver 1.3 Performance Projects 

Performance 
Items a=.85 
SB Projects 
a=91 



Class Work 
a=.93 



Driver 1 .4 Student Projects 

Student Projects 

[Subject Centered 

Representation] 

a=.42 

Driver 1 .5 Subject Centered Classwork 

Subject Centered 
a=.60 



Description of Factor Components 



Teacher observed Modeling Mathematics Connections; 
Guiding Mathematics Communication; Modeling 
Mathematics reasoning; and Guiding students in making 
Representations. 

Teachers observed Guiding students with making 
Connections and using Visual connections in room. 

Teachers observed fostering student centered 
communication and reasoning using Subject centered 
reasoning. 

Teacher observed Guiding students and Modeling problem 
solving (PS). 

Teacher gives and counts homework and the type of 
homework activities. 



Use calculators or computers as instructional activities; 
Build models or charts. 

Demo, presentation, or proof; Use measuring tools; 
Measure Objects; Analyze data for conclusions while PS; 
(Solve word problems from text or worksheet). *’* 

Complete problems from textbooks or worksheets; Do 
arithmetic computations for homework. 

Use hands-on materials or manipulatives such as counting 
blocks and algebraic tiles. 

Student Demonstrations; Maintaining Portfolios; Working 
on Projects. 

Collect or Analyze data; solving problems; Collect or 
Analyze via Internet; Projects outside the classroom; 
Project lasting longer than a week; write a report; explain 
reasoning; Use graphing calculators; (Solve problems in 
groups). 

Do arithmetic procedures; Do computations from text or 
worksheet; Show steps in problem solving; Read about 
math (non-text); Work with hands-on materials; Work in 
groups; Written assignment for text or worksheet; Use 
hands-on materials; Write problem solving explanations; 
(Use computer tutorial software); (Solve novel problems). 

Participate in Projects outside the classroom; Project 
lasting longer than a week. 

Teacher observed using Subject and teacher centered 
representation. 



Teacher observed using Subject centered communication, 
problem solving, and connections. 



Source 



Standards- Based 
(SB) Checklist of 
Classroom 
Observ'ations 

SB Checklist of 
Classroom 
Observations 
Analysis of 
Classroom 
Observation 
SB Checklist of 
Classroom 
Observations 
Student Survey of 
Classroom Practices 
(SCP) 



Teacher Survey of 
Classroom Practices 
(SCP) 

Teacher SCP 
Instructional 
Activities 

Student SCP 
Student SCP 



Teacher SCP 
Assessment Practices 
Teacher SCP 
InSructional 
Activities 



Teacher SCP 
Instructional 
Activities 



Student SCP 

Analysis 
of Classroom 
Observation 



Analysis of 

Classroom 

Observation 



Cronbach’s Alpha 

Negative driver model factors are enclosed in brackets 
Negative factor components are enclosed in parentheses 
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Driver 


Driver Name 


Factor Name 


Description of Factor Components 


Source 






[Classroom 


Work individually in class; Estimate, predict, or guess 


Teacher SCP 






Activities] 


while PS; Take tests; Work in groups to improve work; 


Instructional 






a=.6I 


Watch teacher demonstrate PS; Use manipulatives; Collect 
data using hands-on materials; (Presentation about 
concepts or projects); (Take a test on a computer); (Use 
Portfolios); (Apply PS to the real world). 


Activites 


Driver 2 


Unified Policy 








Driver 2. 1 


Instructional 


Students and 


Meeting the needs of students and parents and preparing 


Teacher SCP 




Influences 


Parents a=.59 


for the next grade. 


Influence 






Standards 


District, state, text, or national curricula; Pre-service 


Teacher SCP 






a=.86 


experience; District or state tests. 


Influence 






Goals and 


Quality of math and science goals; Measurable 


School Improvement 






Mission a= 79 


Assessments; Mission statement; Staff development plan. 


Plan 


Driver 2.2 


Accountability 


Teacher 


Impact of accountability; Impact of SB instruction; 


Principal Interview 




and Standards 


Accountability 

a=.45 

Impact of USI and 


(Impact community resources). 






Impact of USI on science achievement; Impact of 


Principal Interview 






Standards a=.88 


Standards on science instruction; (Impact resource 
coordination). 








Impact of USI PD 


Principals’ rating of the impact of math and science PD 


Principal Interview 






a=.60 


and USI impact on math Achievement. 








Technology 


Principals’ rating of USI Impact on tech nology and the 


Principal Interview 






a=.83 


Impact on Achievement. 




Driver 3 


Unified Resources 








Driver 3. 1 


Equity 


Equity 


Encourage minorities and females; Teach, estimation, at 


Teacher SCP 






a=.90 


assigned level, problem solving, diverse abilities, & varied 
cultural. 


Preparation 






Use S-B Practices 


Meet standards; Varied assessments; Manipulatives; Adapt 


Teacher SCP 






a-89 


curricula; Integrate math with other subjects. 


Preparation 


Driver 3.2 


PD Time and 


Prof Dev. Hours 


Total time spent in PD in both content and pedagogy. 


Teacher SCP PD 




Coursework 


a=.85 

Math Courses 
a=86 


Number of Math and Math Education courses. 


Teacher SCP PD 






Standards-Based 


Technology; Needs of students; Journals; Multiple 


Teacher SCP PD 






Instruction a=.84 


Assessment. 




Driver 3.3 


PD on SB 


SB Activities 


Portfolios; Networks; New methods; Math content. 


Teacher SCP PD 




Activities 


a=.77 










Special Needs 


Teach students with LEP, LD, or Physical Disabilities. 


Teacher SCP 






a=.8I 




Preparation 


Driver 4 


Stakeholders 








Driver 4.1 


Community 


Family 


Parent & stakeholder involvement; Mobility; and 


Evaluation of School 






a=.66 


attendance based on family. 


Improvement Plans 






Community 


SIP parents; Community involvement; Belief all children 


Evaluatbn of School 






a=66 


can learn; School demographics; (School Partnerships). 


Improvement Plans 






Achievement 


Increased achievement; Advanced courses; Plan to 


Evaluation of School 






a=.69 


improve Math/science. 


Improvement Plans 


Driver 5 


Achievement 








Driver 5. 1 


Math Scores 


1996 Math 
Achievement 


Math Achievement Scores. 


Math96 






1997 Math 


Math Achievement Scores. 


Math97 






Achievement 
1998 Math 
Achievement 


Math Achievement Scores. 


Math95 






1996 Math 


Math Achievement Scores. 


Math96 






Achievement 
1998 Math 
Achievement 


Math Achievement Scores. 


Math98 


Driver 5.2 


Math Gain 


1995-97 Change 


Change in Math Gain Scores. 


MathDiff9597 






1995-96 Change 


Change in Math Achievement Scores. 


MathDif9596 






1995-99 Change 


Change in Math Achievement Scores. 


MathDif9599 
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Driver 


Driver Name 


Factor Name 


Description of Factor Components 


Source 


Driver 6 


Achievement Gap 








Driver 6. 1 


H-W-F 


Hispanic Change 


Change in Math Achievement Scores for Hispanics. 


DifHmaths95-99 






White Change 


Change in Math Achievement Scores for Whites. 


DifWmaths95-99 






Female Change 


Change in Math Achievement Scores for Females. 


DifFmaths95-99 


Driver 6.2 


M-B 


Male Change 


Change in Math Achievement Scores for Males. 


DifMmaths95-99 






Change 


Change in Math Achievement Scores for Blacks. 


DifBmaths95-99 


Driver 7 


School Culture 








Driver 7. 1 


School Culture 


Vision 


Faculty and staff see themselves as having a shared vision. 


School Culture 
Quality Survey 
(SCQS) 






Teamwork 


Working together cooperatively in an atmosphere of 
mutual respect and caring. 


SCQS 






Facilitative 


Faculty and staff view the administration of the school to 


SCQS 






Leadership 


be providing facilitative leadership. 








Learning 


Eagerly investing, learning and working together toward 


SCQS 






Community 


their shared goals. 




Driver 7.2 


Teacher Opinions 


Culture of 


Share; Learn new things; Make decisions; Supported. 


Teacher SCP 






Sharing 




Opinions 






Standards- Based 


Have curriculuar materials; All students and learn; peer 


Teacher SCP 






Beliefs 


planning; Observe teachers. 


Opinions 






[Traditional 


Basic facts; basic skills; repeated practice. 


Teacher SCP 






Beliefs]* 




Opinions 
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Table 2. Correlation matrix of standardized scores 
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Table 3b. Seven Driver Mode): Total effects o f exogenous variables (policy, resources, stakeholders, culture, and instruction) on endogenous variables (achievement and instruction) 
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Figure 1 . Structural Equation Model of Student Achievement Gains 
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Figure 2. Graph of Combined Elementary, Middle, and High Schools 



1996 through 1999 Combined Elementary, Middle, & High School 
Standardized Math Mean Scores for USI and Non-USI Schools 
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Figure 3. Graph of Elementary Schools 



1996 through 1999 Elementary School-Level Standardized Math Mean 
Scores for USI and Non-USI Schools 
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Figure 4. Graph of Middle Schools 
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Figures. Graph of High Schools 



1996 through 1999 High School-Level Standardized Math Mean 
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